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ABSTRACT 

Today,  the  US  Navy 's  submarine  fleet  is  equipped  with  a  multitude  of  advanced,  highly  capable  acoustic 
sensors  and  systems  that  are  utilized  in  many  aspects  of  tactical,  navigational,  and  operational  Command 
and  Control  (C2).  These  modern  sonar  systems  collect,  process,  store,  disseminate,  and  display  sensor 
data.  Rapid  advances  in  processing  techniques,  computing  power,  and  sensor  technology  present  a  new 
challenge  -  sonar  operators  require  extensive  training  and  experience  in  order  to  deal  with  an  enormous 
amount  of  acoustic  data  using  2D  display  and  interaction  tools,  which  have  remained  nearly  unchanged  in 
the  past  25  years.  Fraunhofer  CRCG  and  the  Naval  Undersea  Warfare  Center  present  a  highly  interactive 
Large  Scale  Visualization  Environment  (LSVE)  for  performing  the  essential  tasks  of  detection  and 
classification  of  sonar  contacts.  Our  prototype  application  employs  a  semi-immersive  3D  display  system, 
multiple  and  mixed  modalities  of  interaction  and  feedback,  and  state-of-the-art  volumetric  visualization 
techniques.  An  operator  can  use  the  LSVE  to  rapidly  detect  a  contact  in  a  low  signal  to  noise  environment 
and  perform  the  tasks  of  detection,  tracking,  and  classification  of  contacts. 

Index  Terms  -  Data  Visualization,  Sonar,  Interactive  Data  Analysis. 


INTRODUCTION 

Submarine  sonar  system  technology  has  benefited  in  recent  years  primarily  from  accelerated 
developments  in  digital  signal  processing,  computational  power,  and  advances  in  sensors.  Today’s  most 
capable  undersea  platforms  deploy  several  different  acoustic  sensor  arrays  both,  hull-mounted  and  towed. 
Each  sensor  suite  is  specifically  designed  for  a  particular  mission,  creating  a  massive  amount  of  sensor 
data  to  be  analyzed  by  the  sonar  operators.  Even  though  these  sonar  arrays  and  processing  systems  are 
state-of-the-art,  the  presentation  of  acoustic  data  to  the  operator  onboard  these  platforms  has  not 
appreciably  changed  in  the  last  35  years. 

The  analysis  of  relatively  low-level  sensor  data  by  an  operator  is  unique  today  to  sonar  Operators  must 
search  and  evaluate  acoustic  energy  across  many  frequency  bands,  apertures,  arrays  and  directions. 


Paper  presented  at  the  RTO  1ST  Workshop  on  “Massive  Military  Data  Fusion  and  Visualisation:  Users  Talk 
with  Developers”,  held  in  Halden,  Norway,  10-13  September  2002,  and  published  in  RTO-MP-105. 
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and  make  decisions  on  the  presence  or  absence  of  contacts  of  interest,  largely  unassisted  by  automation. 
The  sonar  operator  is  challenged  by  an  enormous  data  overload  problem,  coupled  with  antiquated, 
non-intuitive  display  technology  with  restricted  interaction  and  limited  tools  to  accomplish  the  vital  tasks 
of  detection,  tracking,  and  classification  of  contacts. 

To  maximize  performance  under  these  conditions,  the  conduct  of  undersea  warfare  requires  the  highest 
degree  of  information  extraction  and  exploitation  from  a  myriad  of  onboard  acoustic  sensors  and 
processing  systems.  A  new  technology  and  human  systems  approach  is  needed  in  order  to  focus  high- 
performance  computing,  display,  and  human/machine  interaction  technologies  on  the  problems  and  tasks 
faced  by  human  operators  and  decision  makers. 


INTERACTIVE  APPLICATIONS  FOR  SONAR  DATA  PROCESSING 

In  collaboration  with  the  Naval  Undersea  Warfare  Center  (NUWC)  in  Newport,  Rhode  Island, 
the  Fraunhofer  Center  for  Research  in  Computer  Graphics  (CRCG),  Inc.,  has  taken  the  approach  of 
developing  intuitive,  highly  interactive  analysis  tools  to  increase  the  performance  of  the  sonar  operator. 


Interactive  Data  Exploration 

At  the  most  basic  level,  the  challenge  from  an  interaction  point  of  view  is  to  provide  access  and 
manipulation  tools  to  find  correlations  that  are  hidden  in  the  vast  amount  of  data  available  to  the  user. 


EZ-Grams  -  Exploiting  Harmonic  Relationships 

By  focusing  on  perceptual  information  cues,  NUWC  and  Fraunhofer  CRCG  have  developed  interactive, 
intuitive  tools  and  techniques  by  which  the  operator  can  rapidly  extract  pertinent  information  from  the  raw 
data  within  a  specific  task  domain.  Collectively,  these  display  tools  are  represented  as  a  new  class  of 
aids  called  EZ-grams,  which  help  an  operator  transform  raw  sensor  data  into  useful  information 
[Barton,  Rowland  et  al.  2000].  An  EZ-Gram  embodies  a  collection  of  pertinent  sensor  data  that  represents 
the  hypothesis  that  a  contact  is  of  interest.  EZ-Grams  allow  operators  to  rapidly  test  hypotheses,  search  for 
corroborating  data,  and  build  confidence  in  the  solution. 

One  such  EZ-Gram  is  the  harmonogram,  which  is  based  upon  an  hypothesis  that  energy  of  interest  exists 
in  some  or  all  of  the  harmonics  associated  with  a  given  or  selected  frequency  band,  or  “bin.” 
The  frequency  bins  of  the  harmonogram  are  superimposed  to  enhance  the  dynamic  representation  of 
Doppler,  while  reducing  non-correlated  data  in  the  display.  Furthermore,  the  harmonogram  displays  time- 
record  data  through  animation,  allowing  the  user  to  “replay”  local  history  and  view  target  or  contact 
motion,  which  can  provide  valuable  insight  into  signal  dynamics.  Subtle  effects  not  observed  in  single 
image  data  presentations  are  potentially  quite  obvious  when  animated. 

Figure  1  shows  a  typical  time-frequency  or  “waterfall”  representation  of  acoustic  sensor  data  presented  to 
a  sonar  operator  today,  in  which  data  correlations  are  difficult  to  discern.  In  most  cases,  each  gram  has 
time  on  the  vertical  axis,  and  either  bearing  or  frequency  along  the  horizontal.  Signal  intensity  is 
monochrome  and  subsampled  down  to  only  8  levels,  i.e.  3  bit  resolution  presentation.  These  displays  are 
the  primary  workplace  for  the  sonar  operators. 
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Figure  1:  Typical  Time-History  Spectrogram  Sonar  Display. 

Figure  2  shows  a  harmonogram  in  which  a  selected  hypothesis  is  clearly  shown  to  be  valid,  because  it  is 
displayed  as  a  harmonically  related  frequency-frequency  plot,  with  time  history  presented  to  the  user  as  an 
animated  sequence.  Note  that  only  the  data  already  available  in  the  waterfall  is  portrayed  in  the 
harmonogram,  but  the  data  is  filtered  to  the  hypothesis  of  interest,  and  displayed  so  that  the  information 
cues  that  the  operator  is  seeking  are  distinctly  presented.  NUWC  is  currently  developing  several  examples 
of  EZ-Gram  display  tools  for  transition  into  current  and  future  US  Naval  submarine  sonar  display  systems. 
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Figure  2:  Harmonogram  Representation. 


Large  Scale  Visualization  Environment 

Fraunhofer  CRCG  and  NUWC  have  used  a  two-stage  approach  in  the  development  of  an  intuitive,  highly 
interactive.  Large  Scale  Visualization  Environment  (LSVE)  that  will  enable  human  talents,  experience. 
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and  intuition  to  leverage  and  direct  high-performance  computing  resources  [Encarnagao,  Barton  et  al. 
2000].  The  LSVE  consists  of  two  main  components,  “search&detect”  and  “analyze&classify”  which 
will  be  discussed  below.  The  LSVE  has  been  implemented  on  a  “virtual  table”  display  device  as  seen  in 
Figure  3.  The  rear-projection  display  surface  can  be  oriented  at  any  angle  from  horizontal  to  nearly 
vertical. 


Figure  3:  System  Setup  (above).  The  Two-Handed  Input 
Device  -  Pen  and  Pad  -  for  the  Virtual  Table  (below). 

Transparent  pen  and  pad  props  made  of  Plexiglas  have  been  created  to  display  stereoscopic  3D  graphics 
that  the  user  perceives  as  coincident  with  the  physical  position  of  the  devices  [Schmalstieg,  Encarnagao  et 
al.  1999].  The  transparent  non-reflecting  surface  of  the  pad  virtually  disappears  when  the  table  graphics 
are  switched  on.  The  user  then  sees  computer-generated  images  while  the  physical  properties  of  pen  and 
pad  are  retained.  In  particular,  pen  and  pad  allow  tactile  feedback  and  users  intuitively  understand  and  can 
easily  exploit  the  relative  position  and  orientation  of  pen,  pad,  and  table  surface. 

Objects  can  be  aligned  with  the  surface  of  the  pad,  and  also  appear  floating  above  or  below  the  pad. 
The  pad  essentially  becomes  a  hand-held  palette  that  may  carry  all  kinds  of  2D  and  3D  user-interface 
elements  (e.  g.,  buttons,  sliders,  dials)  as  well  as  three-dimensional  objects  [Encarnagao,  Schmalstieg  et  al. 
2000].  The  pen  is  equipped  with  a  single  button  that  triggers  various  actions  depending  on  the  context  in 
which  it  is  used. 
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Data  Management 

The  data  sets  examined  in  this  effort  were  based  on  a  Spatial  Vernier  (range  focused)  simulation  of  the 
sonar  space.  They  consisted  of  one  broadband  data  set  and  four  bands  of  narrowband  data.  The  broadband 
dataset  is  a  four-dimensional  dataset  consisting  of  the  dimensions  time,  range,  beam  and  power.  The  time 
dimension  consists  of  7000  uniformly  sampled  bins.  The  beam  dimension  has  360  bins,  each  equivalent  to 
an  angular  degree.  The  range  bins  consist  of  a  eight  focus  distances  between  2500.0  and  12000.0  yards. 
Therefore  a  signal,  if  it  exists,  will  be  evident  in  all  of  these  ranges  but  focused  in  one.  The  narrow  band 
datasets  were  derived  from  an  identical  simulation  to  the  broadband  data  set.  The  difference  is  the 
narrowband  data  provides  information  on  the  frequency  of  the  signal.  The  narrowband  data  set  is  therefore 
five-dimensional  and  is  broken  into  four  bands  of  frequency  (A-D)  between  6.0  and  450.0  Hz,  which  also 
use  a  different  number  of  beams  (between  45  and  360).  The  number  of  frequency  bins  in  each  dataset 
is  191. 

Search  &  Detect  Component 

The  LSVE  first  component  (“search&detect”)  aims  at  providing  advanced  visualization  and 
human/machine  interface  techniques  to  enable  sonar  operators  to  quickly  and  confidently  detect  and 
classify  contacts  in  low  signal-to-noise  dataspaces.  This  can  be  thought  of  as  a  problem  of  searching  for 
subtle  contact  indicators  in  a  very  large  and  multidimensional  dataspace.  The  goal  is  to  enable  the  operator 
to  focus  computational  resources  on  the  volumes  of  interest,  providing  rapid  initial  detection. 

The  dataspace  is  represented  as  a  set  of  4D  volumes,  that  is,  3D  volumes  that  are  dynamically  updated 
during  the  time  course  of  an  operation.  Multi-sensory  visualization  techniques  are  employed  to  indicate 
volumes  of  interest,  which  may  contain  a  contact  or  contacts. 

The  operator  is  initially  presented  with  a  coarse  visualization  of  potential  areas  of  interest. 
This  visualization  employs  a  low-resolution  visual  presentation  for  fast  searches  over  the  data  space 
assisting  the  user  with  the  decision  on  where  to  focus  attention.  The  visual  presentation  of  the  dataset 
appears  subdivided  in  regions  with  a  cubic  shape  or  ‘cells’.  The  smaller  a  cell  is  subdivided 
(more  resolution)  the  more  interesting  that  volume  is  to  be  explored,  according  to  the  chosen  analysis 
algorithm.  Within  each  cell  a  cloud  of  points  is  displayed  representing  information  about  the  signal 
intensity  within  that  region.  The  number  of  points  as  well  as  the  color  intensity  is  proportional  with  the 
signal  intensity.  Dense  white  clouds  represent  higher  signal  above  threshold  than  sparse  dull  color  clouds. 

For  the  exploration  of  the  data  set,  numerous  interaction  techniques  are  supported,  based  on  the 
transparent  interaction  props  (pen  and  pad)  described  earlier.  Voice  input  is  also  supported. 


Analyze  &  Classify  Component 

In  the  second  LSVE  component  (“analyze&classify”),  we  utilize  a  highly  interactive  3D  representation  of 
the  tactical  environment.  Once  an  area  of  interest  has  been  designated  in  the  initial  search  phase, 
the  visualization  coverts  to  a  volumetric  display  of  the  multidimensional  raw  data  of  the  corresponding 
passive  narrowband  sensor  information.  Employing  visual  and  auditory  cues,  the  operator  can  then  interact 
with  and  interrogate  this  data  cloud.  Lastly,  we  provide  aids  to  display  non-spatial  qualitative  relationships 
in  this  data  set,  in  an  attempt  to  capitalize  on  inherent  human  pattern  recognition  capabilities. 

At  any  point  in  time,  an  area  of  interest  can  be  selected  and  then  further  investigated.  In  analyze&classify 
the  visualization  switches  to  the  raw  data  view  (Figure  4)  of  the  corresponding  passive  narrow-band  sensor 
information.  The  operator  can  further  apply  many  interaction  and  manipulation  techniques  for  volume 
data,  most  of  which  have  been  successfully  applied  in  the  past  to  the  area  of  medical  visualization. 
These  include  defining  oblique  and  axis-parallel  cutting  planes  to  detect  tracks,  maximum  intensity 
projection  to  enhance  the  contrast,  and  adjusting  the  intensity  lookup  table  for  certain  known  thresholds. 
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Figure  4:  ‘Analyze&Classify’  Visualization:  Detection  of  multiple  threats  through  modification  of 
the  volumes  lookup  table  (left)  and  employing  an  additional  harmonogram  (right). 

In  the  real  world  we  analyze  objects  by  grabbing,  rotating  and  bringing  them  closer  to  our  eyes  to  get  more 
details  about  interesting  parts.  We  provide  the  same  abilities  in  LSVE  with  our  interaction  tools. 

While  no  formal  evaluations  have  been  conducted,  users,  including  active  or  retired  Naval  personnel  with 
USW  experience,  were  impressed  by  the  LSVE. 


DECISION-CENTERED  VISUALIZATION 

Often  under  high-stress  and  workload  in  high-risk  situations,  the  SONAR/Control  team  must  rapidly 
achieve  and  maintain  situation  awareness  -  they  must  understand  the  tactical  situation  within  the  sensor 
performance  envelope.  In  such  situations,  operators  and  decision  makers  do  not  have  the  luxury  of  time 
either  to  interactively  specify  an  information  presentation,  or  describe  to  a  human  operator  what  to  display 
and  how  to  display  it. 

Current  work  in  the  visualization  community  supports  highly  interactive  and  often  immersive 
presentations  in  a  variety  of  applications.  While  extensive  interactive  viewing  and  data  selection  is 
supported,  current  visualization  systems,  for  the  most  part,  only  maintain  information  about  pixels  and 
polygons.  They  do  not  represent  and  use  meta-knowledge  about  the  data  and  information  that  is  being 
displayed,  and  thus  have  no  capacity  for  actively  supporting  the  user  by: 

•  presenting  information  known  to  be  critical  for  task  performance, 

•  alerting  the  user  to  missing  data,  and 

•  highlighting  information  crucial  to  sound  decision  making. 

A  new  technology  and  systems  approach  is  needed  which  will  provide  the  SONAR/Control  team  with 
interactive  information  visualizations  that  merge  decision  support  with  situational  awareness. 
An  important  approach  to  this  problem  is  to  embed  in  the  visualization  system  knowledge  of  the 
application  domain,  the  key  concepts  and  operations,  its  missions,  tasks,  and  decisions. 

Fraunhofer  CRCG  has  developed  a  new  technology  called  Decision-Centered  Visualization  (DCV),  which 
can  provide  decision  makers  with  interactive  information  visualizations  that  merge  domain  knowledge  and 
decision  support  with  situational  awareness  presentations.  Please  see  Figure  5.  Decision  and  task  models, 
knowledge  of  the  information  environment  of  the  application  domain,  including  concepts,  events, 
and  operations,  are  tightly-coupled  with  the  human/machine  interface  and  the  visualization  architecture  in 
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order  to  produce  timely,  decision-centered  visualizations.  DCV  uses  knowledge  to  track  the  context  of  an 
evolving  situation,  to  classify  and  prioritize  incoming  operational  data,  and  to  automatically  tailor  and 
organize  presentations  in  context  with  the  tasks  and  decisions  being  worked,  enabling  a  genuine 
human/machine  information  symbiosis.  The  main  objectives  of  DCV  are  to  reduce  the  information 
workload  of  operators  and  decision  makers  by  providing  in-context,  to-the-point  information 
presentations,  and  to  reduce  decision  time  by  timely  presentation  of  mission-relevant  and  mission-critical 
information. 


Domain 

Abstractions 


Agents  Monitor 
&  Alert  V''  ' 


Decision 

Requirements 

^  J 


Figure  5:  High-level  Architecture  for  Decision-Centered  Visualization  (DCV).  The  distinguishing 
feature  of  this  technology  is  the  incorporation  of  knowledge  modules  into  the  visualization 
system.  These  knowledge  modules  consist  of  domain  ontology  and  domain 
abstractions,  decision  requirements  and  decision  focus. 


The  DCV  architecture  is  intended  to  be  domain-independent  to  support  visualization  in  a  variety  of 
applications.  Application-specific  information  is  localized  in  well-defined  modules  -  domain  ontology, 
domain  abstractions,  and  decision  requirements  -  that  are  addressed  early  in  the  design  phase  when 
developing  a  particular  application.  The  DCV  interaction  cycle  is  intended  to  be  generic  and 
parameterized  in  order  to  display  and  manipulate  this  application-specific  information. 


CONCLUSIONS  AND  FUTURE  WORK 

We  have  presented  a  variety  of  tools,  approaches  and  paradigms  in  support  of  effective  interactive  data 
analysis  for  the  underwater  sonar  space.  Furthermore,  Fraunhofer  CRCG  is  currently  investigating  the 
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integration  of  knowledge  engineering  and  visualization  technology,  to  enable  Decision-Centered 
Visualization 
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SYMPOSIA  DISCUSSION  -  PAPER  NO:  11 


Author’s  Name(s): 

Cdr.  Robert  J.  Barton,  NUWC,  Newport,  USA 

Mr.  David  Zeltzer,  Fraunhofer  Center  for  Research  in  Computer  Graphics  Inc.,  USA 

Question: 

How  are  the  graphics  displayed  on  the  tablet? 

Author’s  Response: 

There  are  not  any  actual  graphics  on  the  tablet.  The  virtual  reality  environment  can  sense  where  the  user  is 
looking  and  can  project  the  graphics  to  a  location  on  the  virtual  table  to  make  it  seem  like  graphics  are  on 
the  tablet. 

Comment: 

Experience  can  be  incorporated  in  the  development  cycle  when  the  rules  are  being  implemented. 

Question: 

Has  historical  data  been  incorporated?  Can  the  user  ask  for  retrospective  information? 

Author’s  Response: 

As  this  is  a  prototype,  that  has  not  yet  been  developed,  however  it  would  be  important  in  an 
implementation  of  the  system. 
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SONAR  SDC  Visualization 

•  Multi-Dimensional,  Low 
Signal-to-Noise  Data  Space 

-  Range,  Bearing, 

Frequency,Tinie 

•  Many  Sensors  ^  Data 
Overload 

•  Since  End  of  Cold  War, 

Naval  Operations  in  Very 
Noisy  Littoral  Waters 

•  Is  There  a  Better  Way  to 
Visualize  SONAR  Data? 
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1.  Rapid  Scan  for 
Volumes  of  Interest 


2.  Select  Volumes  of  Interest 
for  Abstract  Broadband  Visualization 


5.  Select  Another 
Volume  of  Interest 


GIST  Harmonogram 


LSVE  USW  Visualization 
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3.  Multimodal  and 
Multimedia  for  Detailed 


Analysis  of  Abstract 


4.  Apply 
Visualization 
Tools  &  “Grams” 
to  Examine  Actual 
Data 
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Interactively  Rotate  and  Zoom 
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Interactively  View  and  Manipulate  Actual  Data 


Time  fO-llOl 


Bearing  (O-^J 
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3D  Interaction  Technology 
Virtual  Table 
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Decision-Centered  Visualization 
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Background 

•  Environmental  Complexity,  Fog  of  War 

-  Littoral  environments  -  sea,  land,  air 

-  Friendly,  hostUe  &  neutral  forces;  air/surface/water  craft 

•  Operational  Restrictions  -  ROE 

•  Speed  of  Change 

-  High  speed  weapons 

-  Rapid,  coordinated  force  deployment 

•  High  Volume  Data  from  Multiple  Sources  and  Sensors 

•  Result  —  Information  Overload! 
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Take-Away  Message 


Information  Overload? 


Use  Agents  and  Knowledge  to 
•  Anticipate  What  People  Need  to  Know 
•  Build  Integrated,  Intuitive  Displays 
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DCV  Concept 

:  I  Mission  Info 


Area  of  Ops  Info 


Anticipate  Friendlies 

Threats 


Display 


Warn  & 
Alert 


Missing  Data 
Uncertain  Data 
Time-Critical  Data 
Mission-Critical  Data 
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DCV 
Architecture 
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Warfare  Domain  Ontology 


Domain 

Abstractions 


Agents  MonI  or 
&  Alert! 


Association 

Engine 


Classify,  Prioritize, 
Associate  Incoming 
Data 


Domain 

Ontology 


Decision 

Requirements 


What  Are  the 
Things  and  Actions 
in  the  World? 

How  Do  They 
Relate  to  Each 
Other? 
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Domain  Abstractions 


Domain 

Abstractions 


Monitor 
&  Alert 
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Engine 
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Decision 

Focus 


Classify,  Prioritize, 
Associate  Incoming 
Data 


Domain 

Ontology 


Decision 

Requirements 


Application-Specific 
Instances  of  Ontology 
Concepts 
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Infantry  SA  Domain  Abstractions 


•  Unit  type 

•  Home  base 

•  Current  location 

•  Readiness 

•  Resources 

•  Capabilities 

•  Strength 

•  Logistic  support 

•  Combat  support 

•  Weapons 

•  Equipment 

. .  J 


Division 
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Brigade 
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Battalion 
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Company 

Platoon 

r 

Squad 

Doctrinal  Task 
Organization 


CRCG 


Fraunhofer 


Center 
for  Research 

in  Computer  Graphics  Inc. 


11-16 


Decision  Requirements 


Domain 

Abstractions 


Agents  Monitoi 
&  Alert 


Association 

Engine 


H 
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Decision 

Focus 


Decision 

Requirements 


Classify,  Prioritize,  | 
Associate  Incoming  | 

Data  I 

▼ 

f  ^  \ 

Domain 

Ontology 


For  a  Given  Decision, 
Information  Needed  to  Make 
a  Sound  Decision 
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Decision  Focus 


Domain 

Abstractions 


Decision 

Requirements 


Computer  Inference  of 
Intentions  from  Behavior 
Remains  “Grand  Challenge” 


Classify,  Prioritize,  | 
Associate  Incoming  | 

Data  I 

▼ 

f  ^  \ 

Domain 

Ontology 


For  DCV,  Decision  Maker 
Explicitly  Specifies  Current 
Mission  Event(s),Task(s)  & 
Decision(s) 
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Prototype  Decision  Focus  GUI 


•  Human 

Proposes 

Route 

•DCV 

Suggests 

&  Alerts 


•  Information 
Push/Pull 
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“Smart 
Maps’ 
for 

Infantry 

SA 
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Thank  You! 
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